C R A S H !

The @Baker server suffered a disk crash on April 27.

We're Dazed and Confused, but Trying to Recover

Here's a chronology of what happened:

1999-Apr-27 16:00 EDT
People report that they can't reach @Baker. They are right! The Webmaster has to leave for class, so nothing is done until the next day.

1999-Apr-28 17:00 EDT
The server refuses to reboot. We try that disk in other machines, and other disks in that machine. All signs point to a failed disk. We borrow a PC from the company where one of us works and begin work on a temporary server which will at least let people know what's going on.

1999-Apr-29 02:27 EDT

We have a temporary web server which can display only one page
All mail is being refused. (We're trying to figure out how to reply with a canned message explaining what happened.)
Name services are somewhat disrupted; that'll be fixed by Saturday afternoon.
We have backup tapes, but they're many weeks old.
The crashed disk is being sent off to a data recovery service to see what can be salvaged.
All sign-in information is backed up on a different computer, but the sign-ins will have to be reprocessed by hand if we can't recover from the crashed disk. [It turns out that only most sign-in information is backed up.]

1999-Apr-30 20:11 EDT
So I cheered up, and sure as yer born, things got worse! Your humble Webmaster stepped in a pile of slippery tree seeds, those little propeller-looking thingies, fell, and broke every bone in his body. (I exaggerate. I've only broken three bones and they're all in my right ankle. Surgery is scheduled for Tuesday afternoon.) This will make the recovery process for the server even longer than I had hoped. I still plan to straighten out name services on Saturday; otherwise, Peach Net will "expire our zone" and @Baker will disappear from the Internet. Bad. That's probably all that'll happen until I get out of the hospital. (sigh.) The disk was shipped off to the recovery service on Thursday evening, and I probably wouldn't have done much before finding out the status of that anyway.

1999-May-02 20:19 EDT
Name services (also called DNS) is running on the substitute server. This means that @Baker is once again announcing its presence to the Internet. Good. I had hoped to get some sort of autoresponder running for mail today, but I'm afraid I've run down at sundown. <grin> Probably nothing else will get done before Thursday at the earliest because of hospitals. Federal Express confirms delivery of the crashed disk at DataSavers in California. We'll see what they're able to do with it.

1999-May-05 21:23 EDT
Down, but not de-feeted We haven't heard from the data recovery people yet, but the ankle recovery crew have (mostly) completed their work on your humble Webmaster, and I'm pleased to report that I'm at home and sulking. (Yes, it's true, I will be on crutches for six weeks!)

1999-May-10 17:29 EDT
Today we received word that the disk from @Baker is not recoverable. This means applying a very old backup tape, then rebuilding piece by piece. A new disk was shipped from Seagate in Oklahoma last Thursday and is expected to be delivered tomorrow. I am tempted to wait for delivery of Red Hat Linux 6.0, which started shipping today, before rebuilding our server. By the weekend, you should see the "old" @Baker, but with lots of recovery work to do. [Didn't happen.]

1999-May-15 01:05 EDT
Well, I finally have all the stuff: two new disks, a new Ethernet card (don't even ask!) and a copy of Red Hat Linux 6.0. Tomorrow I can start rebuilding @Baker. Here's where we are:

Server: The box is up and running with new disks. I have to download a bunch of programs such as the Majordomo list manager and get them installed.
Web Stuff: NCSA has quit supporting the Web server we were using. The "new" @Baker will be running Apache. Files will be restored from last July; then I'll start the process of rebuilding by hand. This will be a long and tedious process.
Mail: We'll get the newest sendmail and Majordomo installed. Then all the mailing lists will be cleaned out (since some people may have unsubscribed since the last backup) and people will be invited to resubscribe to mailing lists. Mail sent to Webmaster and not replied to has been irrevocably lost, I'm sorry to say.

That's how things are. We should have something going by next weekend and be on the way to restoring @Baker.

Oh, yeah, the plaster cast is gone and replaced by a black velcro Darth Vader boot. I liked the cast better. There's a row of surgical staples that looks like a zipper down either side of my ankle; looks cool... very Frankenstein. Crutches and no weight-bearing on the right foot for five more weeks. Sigh.

1999-May-16 23:28
The "new" @Baker server is running with Red Hat Linux 6.0; the backup tape has been restored to the second disk so it'll be easy to work with. This page is still being served by the temporary server, thanks to DeKalb Regional Healthcare System. The plan for service restoration (without dates, yet) is as follows:

Establish security on new server (TCP Wrapper, etc.).
Move temporary page (i.e. this page) to new server; move name services to new server.
Install programs not a part of Red Hat Linux, e.g. REXX, SWISH.
Establish Web services with files from backup tape.
Install and test Majordomo.
Re-establish list services; send notification to subscribers.
Start re-processing the sign-ins, a dismaying amount of work.

1999-May-29 17:39
Your 'umble Webmaster is still moving a little slow, but at least we're actually running on the "real" @Baker server, and the temporary server has been removed. We're running Web and name services on the reconstituted @Baker. See above for a description of what work remains.

1999-May-30 18:25
Things are looking up! Majordomo is installed, people who try to go direct to a bookmarked page within @Baker get a meaningful explanation instead of just "404 Not Found" and anyone who tries to send to a Baker mailing list gets a reply explaining what happened. If only I'd been able to get this far in the first week... sigh.

1999-May-31 18:28 EDT
Well! Most of the Web stuff works, and a couple of the mailing lists are working. I'll probably continue to put narrative here as we trudge through the recovery process, but the One True Source of the status of things will move to the What's New page, now that we have it back. things are looking up!

1999-Jun-06 18:03 EDT
The mailing lists are back in operation, but you will have to resubscribe. Next we need to fix the search and sign-in pages.